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Cross Reference to Related Applications 
[0001] The present application claims the benefit of U.S. Provisional Patent 
Application Serial No. 60/506,303 filed September 25, 2003, entitled "Printer Including 
One or More Specialized Hardware Devices," and U.S. Provisional Patent Application 
60/506,302 filed on September 25, 2003, entitled "Printer Including Interface and 
Specialized Information Processing Capabilities," each of which is hereby incorporated 
by reference in its entirety. 

[0002] The present application is a continuation-in-part of the following co-pending 
U.S Patent Applications: Application Serial No. 10/001,895, "(Video Paper) Paper- 
based Interface for Multimedia Information," filed November 19, 2001; Application 
Serial No. 10/001,849, "(Video Paper) Techniques for Annotating Multimedia 
Information," filed November 19, 2001; Application Serial No. 10/001,893, "(Video 
Paper) Techniques for Generating a Coversheet for a paper-based Interface for 
Multimedia Information," filed November 19, 2001; Application Serial No. 10/001,894, 
"(Video Paper) Techniques for Retrieving Multimedia Information Using a Paper-Based 
Interface," filed November 19, 2001; Application Serial No. 10/001,891, "(Video Paper) 
Paper-based Interface for Multimedia Information Stored by Multiple Multimedia 
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Documents," filed November 19, 2001; Application Serial No. 10/175,540, "(Video 
Paper) Device for Generating a Multimedia Paper Document," filed June 18, 2002; and 
Application Serial No. 10/645,821, "(Video Paper) Paper-Based Interface for Specifying 
Ranges CEP," filed August 20, 2003; each of which is each hereby incorporated by 
reference in its entirety. 

[0003] The present application is related to the following U.S Patent Applications: 
"Printer Having Embedded Functionality for Printing Time-Based Media," to Hart et. al, 
filed March 30, 2004, Attorney Docket 20412-8340; "Networked Printing System 
Having Embedded Functionality for Printing Time-Based Media," to Hart et. al, filed 
March 30, 2004, Attorney Docket 20412-8341; and "Multimedia Print Driver Dialog 
Interfaces," to Hull et. al, filed March 30, 2004, Attorney Docket 20412-8454; each of 
which is hereby incorporated by reference in its entirety. 

Background 

Field of the Invention 

[0004] The present invention relates to printing devices and, more specifically, to 
printing devices that can receive music files, generate and deliver a variety of music- 
related paper and electronic outputs. 
Background of the Invention 

[0005] Advances in audio technology have created new opportunities for musicians, 
composers, and music lovers to play, create, and appreciate music. At the forefront of 
these advances has been the advent of MPEG audio layer 3 ("MP3") and related 
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standards for compressing digital audio files. The ability to reduce music files to a 
fraction of their original size has enabled the sharing of literally millions of music and 
other audio files through peer-to-peer networks. While MP3 and other digital audio 
formats are well-suited for providing studio quality recordings, there is still a strong 
demand for other types of musical files - for instance musical scores and Musical 
Instruments Digital Interface (MIDI) files. 

[0006] Scores and MIDI files are particularly useful for composing or writing music. 
Oftentimes, composers will score a musical work or idea soon after its creation, and then 
refine the score as the music develops. MIDI files, because of their small size and ease 
of manipulation, are likewise well-suited to composing, editing, and arranging music. 
MIDI files are also better adapted than MP3s for applications constrained by memory 
limitations. Cellphones, PDAs, and other handheld devices often use MIDI tones as 
signal tones, as do website interfaces and games, in place of bulkier digital audio files. 
In addition, both musical scores and MIDI files often store musical information 
embedded in finished recordings such as the tempo, phrasing, measures, or stanzas of a 
piece, or when a note is played, how loudly, and for how long. This information can be 
useful in marking and indexing finished recordings. 

[0007] Presently, the conversion of audio and music files between different paper, 
digital and analog formats often requires several steps and devices. To convert an analog 
recording into a digital file such as an MP3, and then output versions of the MP3 as a 
musical score and a MIDI file that can be played as a cellphone ringtone requires 
coordination between different systems and outputs. 
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[0008] Thus, there is a need for a unified system that can translate audio files into 
different types of paper and electronic file formats and output the results. 

Summary of the Invention 
[0009] The present invention overcomes the deficiencies and limitations of the prior 
art by allowing users to convert and print their music and audio files to various paper and 
electronic media. In accordance with an embodiment of the invention, a user can send 
an audio or music file in a first format to an audio processing device, and then receive an 
output of the file in a second format. In another embodiment, an audio processing 
device receives a musical score and a music file and indexes the contents of the musical 
file according to positions in the musical score. In an embodiment, there is an apparatus 
for outputting a processed audio/music file. The apparatus comprises an interface for 
receiving audio/music data in a first format, a processor for processing the audio/music 
data, and an output system for outputting the processed audio/music data in a second 
format. 

Brief Description of the Drawings 
[0010] Figure 1 is a block diagram showing an audio processing device in 
accordance with an embodiment of the invention. 

[001 1] Figure 2 is a block diagram of memory of the audio processing device of 
Figure 1 in accordance with an embodiment of the invention. 
[0012] Figure 3 shows an exemplary print dialog interface for use with an audio 
processing device. 
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[0013] Figure 4 is a flow diagram of steps of a preferred embodiment of an audio 
processing device. 

[0014] Figure 5 shows an exemplary document output by an audio processing 
device. 

[0015] Figure 6 is a flow diagram showing a preferred process for retrieving a file 
stored by an audio processing device. 

Detailed Description of the Preferred Embodiments 
[0016] The invention provides various apparati and methods for processing audio 
files to generate a variety of outputs. In one embodiment, a digital audio file is provided 
to an audio processing device 100, converted into a MIDI file and then scored, and the 
resulting audio record is printed out. In another, several versions of a music file are 
provided to audio processing device, and information contained in one version is used to 
create an index to another version. In yet another embodiment, commands to edit and 
output an audio file are received by a printer, carried out, and the result may be output to 
a storage media or network server. In a still further embodiment, a processed audio file 
is broadcast over a playback device installed on a printer or audio processing device 100 
that receives the audio file in unprocessed form over a network. 
[0017] Allowing a user to manage audio and music file conversions with the use of 
embodiments of the invention offers several benefits. First, converting audio data to 
smaller MIDI or paper-based format makes it easier to manipulate the data. In addition, 
the burdens associated with comparing and matching audio files and identifying patterns 
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within the files may be facilitated by the automatic conversion of the files into the 
appropriate format. Finally, the indexing of audio files based on musical segments 
made possible by embodiments of the invention facilitates access to specific portions of 
an audio file. 

[0018] For the purposes of this invention, the terms "audio/music data", 
"audio/music file", "audio/music information" or "audio/music content" refers to any 
one of or a combination of audio or music data. As used herein, the terms "audio data", 
"audio files", "audio information" or "audio content" refer to data containing speech, 
recordings, sounds, MIDI data, or music. The data can be in analog form, stored on 
magnetic tape, or digital files that can be in a variety of formats including MIDI, .mp3, 
or .wav. Audio data may comprise the audio portion of a larger file, for instance a 
multimedia file with audio and video components. As used herein, the terms "music 
files", "music data", "music information" or "music content" means audio data that 
contains music or melodies, rather than pure sounds or speech, and representations of 
such data including music scores or other musical map. Music files can comprise audio 
data that conveys such music or melodies. Music files alternatively can be conveyed for 
instance in a document or graphical format such as Postscript, .tiff., .gif, or jpeg. 
[0019] For purposes of the invention, the audio/music data discussed throughout the 
invention can be supplied to audio processing device 100 in any number of ways 
including in the form of streaming content, a live feed from an audio capture device, a 
discrete file, or as a portion of a larger file. In addition, for the purposes of this 
invention, the terms "print" or "printing," when referring to printing onto some type of 
medium, are intended to include printing, writing, drawing, imprinting, embossing, 
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generating in digital format, and other types of generation of a data representation. 
While the words "document" and "paper" are referred to in these terms, output of the 
system in the present invention is not limited to such a physical medium, like a paper 
medium. Instead, the above terms can refer to any output that is fixed in a tangible 
medium. In some embodiments, the output of the system 100 of the present invention 
can be a representation of audio/music data printed on a physical paper document. By 
generating a paper document, the present invention provides the portability of paper and 
provides a readable representation of the multimedia information. 
[0020] In the following description, for purposes of explanation, numerous specific 
details are set forth in order to provide a thorough understanding of the invention. It will 
be apparent, however, to one skilled in the art that the invention can be practiced without 
these specific details. In other instances, structures and devices are shown in block 
diagram form in order to avoid obscuring the invention. 

[0021] Reference in the specification to "one embodiment" or "an embodiment" or 
the like means that a particular feature, structure, or characteristic described in 
connection with the embodiment is included in at least one embodiment of the invention. 
The appearances of "in one embodiment" and like phrases in various places in the 
specification are not necessarily all referring to the same embodiment. 
[0022] Figure 1 is a block diagram showing an audio processing device or music 
processing printer 100 in accordance with an embodiment of the invention. The audio 
processing device 100 preferably comprises an audio/music interface 102, a memory 
104, a processor 106, and an output system 108. 
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[0023] As shown, in one embodiment, audio/music data 150 is passed through signal 
line 130a coupled to audio processing device 100 to audio/music interface 102 of audio 
processing device 100. As discussed throughout this application, the term "signal line" 
means any connection or combination of connections supported by a digital, analog, 
satellite, wireless, firewire, IEEE 1394, 802.1 1, RF, local and/or wide area network, 
Ethernet, 9-pin connector, parallel port, USB, serial, or small computer system interface 
(SCSI), TCP/IP, HTTP, email, web server, or other communications device, router, or 
protocol. Audio/music data 150 may be sourced from a portable storage medium (not 
shown) such as a tape, disk, flash memory, or smart drive, CD-ROM, DVD, or other 
magnetic, optical, temporary computer, or semiconductor memory. In an embodiment, 
data 150 are accessed by the audio processing device 100 from a storage medium 
through various card, disk, or tape readers that may or may not be incorporated into 
audio processing device 100. Alternatively, audio/music data 150 may be sourced from 
a peer-to-peer or other network (not shown) coupled to the audio/music interface 102 
through signal line 130a or received through signal line 130d, or audio/music data 150 
can be streamed in real-time as they are created to audio/music interface 102. 
[0024] In an embodiment, audio/music data 150 are received over signal line 130a 
from a data capture device (not shown), such as a microphone, tape recorder, video 
camera, or other device. Alternatively, the data may be delivered over signal line 130a 
to audio/music interface 102 over a network from a server hosting, for instance, a 
database of audio/music files. Additionally, the audio/music data may be sourced from a 
receiver (e.g., a satellite dish or a cable receiver) that is configured to capture or receive 
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(e.g., via a wireless link) audio/music data from an external source (not shown) and then 
provide the data to audio/music interface 102 over signal line 130a. 
[0025] Audio/music data 150 are received through audio/music interface 102 
adapted to receive audio/music data 150 from signal line 130a. Audio/music interface 
102 may comprise a typical communications port such as a parallel, USB, serial, SCSI, 
Bluetooth™/IR receiver. It may comprise a disk drive, analog tape reader, scanner, 
firewire, IEEE 1394, Internet, or other data and/or data communications interface. 
[0026] Audio/music interface 102 in turn supplies audio/music data 150 or a 
processed version of it to system bus 1 10. System bus 1 10 may represent one or more 
buses including an industry standard architecture (ISA) bus, a peripheral component 
interconnect (PCI) bus, a universal serial bus (USB), or some other bus known in the art 
to provide similar functionality. In an embodiment, if audio/music data 150 is received 
in an analog form, it is first converted to digital form for processing using a conventional 
analog-to-digital converter. Likewise, if the audio/music data 150 is a paper input, for 
instance a paper score, audio/music interface 102 may be coupled to a scanner (not 
shown) that could be equipped with optical character recognition (OCR) capabilities by 
which the paper score can be converted to a digital output signal like 130a. 
Audio/music data 150 is sent in digitized form to the system bus 1 10 of audio processing 
device 100. 

[0027] In Figure 1, audio/music data 150 is delivered over signal line 130a to audio 
processing device 100. However, in other embodiments, audio/music data 150 may also 
be generated within audio processing device 100 and delivered to processor 106 by 
system bus 1 10. For instance, audio/music data 150 may be generated on audio 
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processing device 100 through the use of music generation software (not shown) for 
composing a MIDI file. Once created on the audio processing device 100, a MIDI file 
can be sent along the system bus 1 10, to processor 106 or memory 104 for instance. In 
another embodiment, audio processing device 100 contains a digital audio recorder (not 
shown) through which live music played on an instrument or output device outside the 
audio processing device 100, for instance, can be recorded. Once captured, digital 
signals comprising the audio recording can then be further processed by the audio 
processing device 100. 

[0028] Commands 190 to process or output audio/music data 150 may be 
transmitted to audio processing device 100 through signal line 130b coupled to audio 
processing device 100. In an embodiment, commands 190 reflect a user's specific 
conversion, processing, and output preferences. Such commands could include 
instructions to convert audio/music data 150 from an analog to digital format, or digital 
to analog, or from one digital format to another, or from a score to music or vice versa. 
Alternatively, commands 190 could direct processor 106 to carry out a series of 
conversions, or to index raw or processed audio/music data 150. In an embodiment, 
commands 190 specify where the processed audio/music data 150 should be output - for 
instance to a paper document, electronic document, portable storage medium, or the like. 
A specific set of commands sent over a signal line 130b to bus 1 10 in the form of digital 
signals instruct, for instance, that audio/music data 150 in a .wav file should be 
converted to MIDI and then scored, and the result burned to a CD. 
[0029] In an embodiment, commands 190 to processor 106 instruct that the 
processed audio/music data 150 be output to a paper document. Preferably commands 
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190 describe the layout of the document 170 on the page, and are sent as digital signals 
over signal line 1 30b in any number of formats that can be understood by processor 1 06 
including page description language (PDL), Printer Command Language (PCL), 
graphical device interface (GDI) format, Adobe's Postscript language, or a vector- or 
bitmap- based language. The instructions 190 also specify the paper source, page 
format, font, margin, and layout options for the printing to paper of audio/music data 
150. Commands 190 could originate from a variety of sources including a print dialog 
on a processing device 160 coupled to audio processing device 100 by signal line 130c 
that is programmed to appear every time a user attempts to send audio/music data 150 to 
the audio processing device 100 for instance. Figure 3 shows one exemplary print dialog 
interface 300 to be displayed for use with an embodiment of the invention. 
Alternatively, commands 190 in the form of responses provided by a user to a set of 
choices presented in a graphical user interface could be sent to processor 106 via a signal 
line 130b or 130d and system bus 1 10 over a network (not shown). A similar set of 
choices and responses could be presented by a hardware display, for instance through a 
touch screen or key pad hosted on a peripheral device coupled to audio processing 
device 100 by a signal line or installed on audio processing device 100. The commands 
maybe transmitted, in turn, to audio processing device 100 through signal line 130b 
connected to the peripheral device or could be directly provided to audio processing 
device 100. In yet another embodiment, conventional software hosted on a machine (not 
shown) could be adapted to solicit processing and output choices from a user and then 
send these to processor 106 on audio processing device 100. This software could be 
modified through a software plug-in, customized programming, or a driver capable of 
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adding "print" options to audio rendering applications such as Windows Media. Various 
possible interfaces for controlling and managing audio/music data are further discussed 
in U.S. Patent Application entitled, "Multimedia Print Driver Dialog Interfaces, 1 ' to Hull 
et. al, filed March 30, 2004, Attorney Docket 20412-8454, which is hereby incorporated 
by reference in its entirety. 

[0030] Although processor 106 of audio processing device 100 of Figure 1 is 
configured to receive processing commands 190 over a signal line 130b, as described 
above, in another embodiment of the invention, processing commands 190 are input or 
generated directly on audio processing device 100. In another embodiment, audio 
processing device 100 does not receive commands at all to process the audio/music data 
150, but contains logic that dictates what steps should automatically be carried out in 
response, for instance, to receiving a certain kind of data 150. For instance, the audio 
processing device 100 could be programmed to convert every.mp3 or .wav file it 
receives to MIDI upon receipt, and then to store the resulting MIDI file to a server on a 
network accessed over signal line 130d. 

[0031] As shown in Figure 1, audio processing device 100 receives audio/music data 
150 and commands 190 over signal lines 130a, 130b and outputs processed audio/music 
data 150 over signal line 130c as a paper document 170 or over signal line 130d as 
electronic data 1 80. Audio processing device 100 may be customized for use with 
audio/music data 150, and may contain various of the modules 200-212 displayed in 
Figure 2 and assorted peripherals (such as an electronic keyboard, microphones) (not 
shown) to generate audio/music data 150. As used herein, the term "module" can refer 
to program logic for providing the specified functionality that can be implemented in 
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hardware, firmware, and/or software. In an embodiment, audio processing device 100 
comprises a printing device that has the capability to generate paper outputs, and may or 
may not have the ability to generate electronic outputs as shown. As used herein, the 
term "printing device" or "printer" refers to a device that is capable of receiving 
audio/music data 150, has the functionality to print paper documents, and may also have 
the capabilities of a fax machine, a copy machine, and other devices for generating 
physical documents. Printing device may comprise a conventional laser, inkjet, portable, 
bubblejet, handheld, or other printer, or may comprise a multi-purpose printer plus 
copier, digital sender, printer and scanner, or a specialized photo or portable printer, or 
other device capable of printing a paper document. In an embodiment, printing device 
comprises a conventional printer adapted to receive audio data, or to output electronic 
data. 

[0032] Audio processing device 100 preferably comprises an output system 108 
capable of outputting data in a plurality of data types. For example, output system 108 
preferably comprises a printer of a conventional type and a disk drive capable of writing 
to CDs or DVDs. Output system 108 may compromise a raster image processor or other 
device or module to render audio/music data 150 onto a paper document 170. In another 
embodiment, output system 108 may be a printer and one or more interfaces to store data 
to non-volatile memory such as ROM, programmable read-only memory (PROM), 
erasable programmable read-only memory (EPROM), electrically erasable 
programmable read-only memory (EEPROM), flash memory, and random access 
memory (RAM) powered with a battery. Output system 108 may also be equipped with 
interfaces to store electronic data 150 to a cell phone memory card, PDA memory card, 
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flash media, memory stick or other portable medium. Later, the output electronic data 
1 80 can be accessed from a specified target device. In an embodiment, output system 
108 can also output processed audio/music data 150 over signal line 130d to an email 
attaching the processed audio/music data 150 to a predetermined address via a network 
interface (not shown). In another embodiment, processed audio/music data 150 is sent 
over signal line 130d to a rendering or implementing device such as a CD player or 
media player (not shown) where it is broadcast or rendered. In another embodiment, 
signal line 130d comprises a connection such as an Ethernet connection, to a server 
containing an archive where the processed content can be stored. Other output forms are 
also possible. 

[0033] Audio processing device 100 further comprises processor 106 and memory 
104. Processor 106 contains logic to perform tasks associated with processing 
audio/music data 150 signals sent to it through the bus 110. It may comprise various 
computing architectures including a reduced instruction set computer (RISC) 
architecture, a complex instruction set computer (CISC) architecture, or an architecture 
implementing a combination of instruction sets. In an embodiment, processor 106 may 
be any general-purpose processor such as that found on a PC such as an INTEL x86, 
SUN MICROSYSTEMS SPARC, or POWERPC compatible-CPU. Although only a 
single processor 106 is shown in Figure 1, multiple processors may be included. 
[0034] Memory 104 in audio processing device 100 can serve several functions. It 
may store instructions and associated data that may be executed by processor 106, 
including software and other components. The instructions and/or data may comprise 
code for performing any and/or all of the functions described herein. Memory 104 may 
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be a dynamic random access memory (DRAM) device, a static random access memory 
(SRAM) device, or some other memory device known in the art. Memory 104 may also 
include a data archive (not shown) for storing audio/music data 150 that has been 
processed on processor 106. In addition, when audio/music data 150 is first sent to 
audio processing device 100 1 10 via signal line 130a, the data 150 may temporarily be 
stored in memory 104 before it is processed. Other modules 200-212 stored in memory 
104 may support various functions, for instance to convert, match, score and map audio 
data. Exemplary modules in accordance with an embodiment of the invention are 
discussed in detail in the context of Figure 2, below. 

[0035] Although in Figure 1, electronic output 180 is depicted as being sent outside 
audio processing device 100 over signal line 130d, in some embodiments, electronic 
output 180 remains in audio processing device 100. For instance, processed audio/music 
data 150 could be stored on a repository (not shown) stored in memory 104 of audio 
processing device 100, rather than output to external media. In addition, audio 
processing device 100 may also include a speaker (not shown) or other broadcasting 
device. An audio card or other audio processing logic may process the audio/music data 
150 and send them over bus 1 10 to be output on the speaker. Not every embodiment of 
the invention will include an output system 108 for outputting both a paper document 
170 and electronic data 180. Some embodiments may include only one or another of 
these output formats. 

[0036] Audio processing device 100 of Figure 1 is configured to communicate with 
processing device 160. In an embodiment, audio processing device 100 may share or 
shift the load associated with processing audio/music data 150 with or to processing 
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device 160. Processing device 160 may be a PC, equipped with at least one processor 
coupled to a bus (not shown). Coupled to the bus can be a memory, storage device, a 
keyboard, a graphics adapter, a pointing device, and a network adapter. A display can be 
coupled to the graphics adapter. The processor may be any general-purpose processor 
such as an INTEL x86, SUN MICROSYSTEMS SPARC, or POWERPC compatible- 
CPU. Alternatively, processing device 160 omits a number of these elements but 
includes a processor and interface for communicating with audio processing device 100. 
In an embodiment, processing device 160 receives unprocessed audio/music data 150 
over signal line 130c from audio processing device 100. Processing device 160 then 
processes audio/music data 150, and returns the result to audio processing device 100 via 
signal line 130c. Output system 108 on audio processing device 100 then outputs the 
result 100, as a paper document 170 or electronic data 180. In another embodiment, 
audio processing device 100 and processing device 160 share processing load or 
interactively carry out complementary processing steps, sending data and instructions 
over signal line 130c. 

[0037] Figure 2 is a block diagram of memory 104 of the audio processor device 100 
of Figure 1 in accordance with an embodiment of the invention. Memory 104 is coupled 
to processor 106 and other components of audio processing device 100 by way of bus 
110, and may contain instructions and/or data for carrying out any and/or all of the 
processing functions accomplished by audio processing device 100. In an alternate 
embodiment, memory 104 as shown in Figure 2 is hosted on processing device 160 of 
Figure 1, or another machine. Processor 106 of audio processing device 100 
communicates with memory 104 hosted on processing device 160 through an interface 
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that facilitates communication between processing device 160 and audio processing 
device 100 by way of signal line 103c. In addition, in embodiments of the invention 
certain elements 200-212 shown in memory 104 of Figure 2 may be missing from the 
memory of audio processing device 100, or may be stored on processing device 160. 
[0038] Memory 104 is comprised of main system module 200, assorted processing 
modules 204-212 and audio music storage 202 coupled to processor 100 and other 
components of audio processing device 100 by bus 1 10. Audio music storage 202 is 
configured to store audio/music data at various stages of processing, and other data 
associated with processing. In the embodiment shown, audio music storage 202 is 
shown as a portion of memory 104 for storing data associated with the processing of 
audio/music data. Those skilled in the art will recognize that audio music storage 202 
may include databases and similar functionality, and may alternately be portions of the 
audio processing device 100. Main system module 200 serves as the central interface 
and control between the other elements of audio processing device 100 and modules 
204-212. In various embodiments of the invention, main system module 200 receives 
input to process audio/music data, sent by processor 106 or another component via 
system bus 110. The main system module 200 interprets the input and activates the 
appropriate module 204-212. System module 200 retrieves the relevant data from audio 
music storage 202 in memory 104 and passes it to the appropriate module 204-212. The 
respective module 204-212 processes the data, typically on processor 100 or another 
processor, and returns the result to system module 200. The result then may be passed to 
output system 108, to be output as a paper document 170 or electronic data 180. 
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[0039] In an embodiment, system module 200 contains logic to determine what 
series of steps, in what order, should be carried out to achieve a desired result. For 
instance, system module 200 may receive instructions from system bus 1 10 indicating 
that the first two measures of a song should be saved to a cell phone card to be played as 
a ringtone based on an .mp3 file of the song. System module 200 can parse these 
instructions to determine that, in order to isolate the first two measures of the song, the 
file must first be converted from a .mp3 file to a MIDI file, then scored, and then the first 
two measures of the MIDI file should be parsed to be output to the cell phone card. 
System module 200 can then send commands to the various modules described below to 
carry out these steps, storing versions of the files in audio music storage 202. 
[0040] Conversion module 204 is coupled to system module 200 and audio music 
storage 202 by bus 110. System module 200, having received the appropriate input, 
sends a signal to conversion module 204 to initiate conversion of audio/music data in a 
first format stored in audio music storage 202 to a file in a second format. Conversion 
module 204 facilitates the conversion between various electronic formats, for instance 
allowing for the conversion among MIDI file, .wav or .mp3 or other digital audio 
formats. As will be understood by those skilled in the art, any number of standard 
software packages could be used, with or without modification, to facilitate such 
conversions including Solo Explorer, freeware dowloadable at 
http://www.perfectdownloads.com/audio-mp3/other/download-solo-explorer.htm or 
Akoff s Music Composer product offered by Akoff Sound Labs at 
http://www.akoff.com/ , (.wav to MIDI conversion software), assorted products offered 
by Lead Technologies of Charlotte, North Carolina (.wav to Windows Media or mp3 
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conversion), or ITunes™ offered by Apple Computer Inc. of Cupertino, California 
(MIDI to mp3/wav conversion). Conversion module 204 may send calls over system bus 
1 10 to these or other software modules to execute the relevant conversion, and direct the 
result to be saved to audio music storage 202. Conversion module may also be coupled 
with hardware to complete specific conversions for instance a digital-to-analog or 
analog-to-digital converter. 

[0041] In another embodiment, conversion module 204 facilitates the conversion of 
an audio file received in analog form to a digital file before it is processed, using an 
analog-to-digital converter for instance. In such a case, conversion module 204 is 
coupled to an analog-to-digital converter, through system bus 110, and activates the 
converter to effect the conversion. In an embodiment, the digital file is returned to 
memory 104 from system bus 1 10, potentially for further processing. In another 
embodiment, conversion module 204 "converts" digital data to audio files. For instance, 
in an embodiment of the invention, audio processing device 100 receives a musical score 
stored in a postscript file sent to it over bus line 1 10. Conversion module 204, equipped 
with optical recognition capabilities for instance, parses the file to obtain the notes, and 
then generates a MIDI approximation using the notes. Standard software such as 
MusicScan sold by Hohner Media of Santa Rosa, CA (score to MIDI conversion) could 
be used or adapted to carry out one or more of these steps. The MIDI file could then be 
converted to a .wav or .mp3 file using the technologies described above. Alternatively, a 
playback module (not shown) could be activated by system module 200. The playback 
module would then retrieve the MIDI file from audio music storage 202 and pass it to 
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system module 200, which would output it to a playback device (not shown) on audio 
processing device 100. 

[0042] Scoring/transcribing module 208 is coupled to system module 200 and 
audio/music storage 202 by bus 110. In an embodiment, scoring or transcription is 
initiated when system module 200 receives instructions to score a digital music file or 
transcribe a speech file stored in audio/music storage 202. Scoring/transcribing module 
208 could access a music file stored in audio/music storage 202 and create a digital file 
that contains a score of the musical notes in the file, for instance in postscript format. 
The postscript file could then be stored in audio/music storage 202. Module 208 could 
also transcribe a digitally recorded audio speech stored in audio/music storage 202, 
resulting in the creation of a file containing a script of the speech. These outputs could 
then be stored in audio/music storage 202 or another location in memory 104 or sent 
over system bus 1 10 to another location on or outside of audio processing device 100. 
To support the musical file to score conversion, any number of standard software 
packages including those offered by Notation Software, Inc. of Bellevue, WA (MIDI to 
score conversion), or Seventh String Software of England (audio recording to score 
conversion) could be used or adapted. The scoring output could be customized to a 
user's needs, and for instance reflect changes in key, tempo, phrasing or other parameters 
automatically performed by the scoring software. Similarly, the transcribing module 
could take live or recorded speech, apply speech recognition technology to the speech 
(such as that offered by Dragon Naturally Speaking 7, made by ScanSoft of Peabody, 
Massachusetts or ViaVoice® offered by IBM of White Plains, New Jersey), and produce 
a text representation of the speech. 
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[0043] Indexing/mapping module 210 is coupled to system module 200 and ( 
audio/music storage 202 by bus 110. In an embodiment, system module 200, having 
received the appropriate input, sends a signal to conversion module 204 to index an 
audio/music file by segment. To carry out this instruction, indexing/mapping module 
210 may access the file on audio/music storage 202 through system bus 110 and parse 
audio data contained in the file into audio segments such as a musical line, bar, stanza, or 
measure, or by song, discrete sound, speech by a speaker, or other segment. The various 
dividers could be determined by indexing/mapping module based on melodic phrasings, 
pauses, or other audio cues. In an embodiment, indexing/mapping module 210 creates a 
new file to store the indexing information and send the new file by system bus 1 10 to be 
stored in audio/music storage 202. In another embodiment, index/mapping module 210, 
responsive to digital commands sent by system module 200, accesses an .mp3 file stored 
in audio/music storage 202 and creates a waveform record of the .mp3 file. The 
waveform can be stored in memory 104 to an electronic document for instance in a 
graphical format that can later be sent to output system 108 to be printed to a paper 
output. Various techniques and interfaces for audio segmentation and audio mapping are 
discussed in more detail in U.S. Patent Application entitled, "Multimedia Print Driver 
Dialog Interfaces," to Hull et. al, filed March 30, 2004, Attorney Docket 20412-8454, 
which is hereby incorporated by reference in its entirety. 

[0044] Matching module 212 is coupled to system module 200 and audio/music 
storage 202 by bus 110. In an embodiment, system module 200, having received the 
appropriate input, sends a signal to matching module 212 to identify the pre-existing 
music file that best matches audio data provided by a user and stored in audio/music 
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storage 202. The audio data to be matched could comprise a portion of a melody. The 
audio data could be sourced by a user recording part of a song on a radio with a digital 
audio recorder or a MIDI file created by a user recalling the riff of a song, for instance. 
In an embodiment, matching module 212 compares the audio data to pre-existing 
recordings or scores and attempts to make a match. Matching module 212 could include 
melody-matching software, for instance GraceNote CDDB or GraceNote MusicID 
provided by Gracenote of Emeryville, California, that has access to a licensed set of 
recordings. The recordings are preferably stored in a database hosted on a networked 
server (not shown). To access the recordings, matching module 212 sends a request to 
system module 200 to fetch the data from the server by way of a signal line, for instance 
an Ethernet connection. Based on data it receives, the melody matching software 
determines which recordings in the database provide the closest match to the audio data. 
In an embodiment, once a match is found, matching module 212 sends a message to 
system module 200 to output to a user a message identifying the matching recording and 
asking if the user would like a copy of the recording. This message could be sent over 
system bus 110 and displayed on an output interface of audio processing device 100 for 
instance. In an embodiment, if the user indicates that she would like a copy of the 
recording, a financial transaction to allow the user to pay for the recording is launched. 
[0045] Figure 3 shows an exemplary print dialog box 300 for use with audio 
processing device 100. The user can input information into the fields of the dialog box 
300 to designate the user's preferences regarding layout, segmentation, etc. The dialog 
box 300 shown could be launched on a graphical display coupled to an audio processing 
device 100 whenever a user selects the print option from an application. Print dialog 
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300 includes some fields that are found in a standard print dialog box such as Printer 
field 304. However, print dialog 300 also displays fields that are not found within 
standard printer dialog boxes, such as Output Options field 314, Advanced Options field 
310, and Preview field 312. As is found in standard print dialog boxes, the top of print 
dialog 300 includes the name (e.g., "Vesoul.mp3") of the audio/music file being printed. 
In Printer field 304, the user can select which printer will carry out the print job, and 
other options with regard to properties of the print job, printing as a image or file, 
printing order, and the like. Additionally, Printer field 304 displays the status of the 
selected printer, the type of printer, where the printer is located, and the like. 
[0046] Output Options field 314 allows the user to choose how she would like the 
audio/music file to be output, and to what media. Input Data Type field 350 is 
automatically populated with the type of file that the user is attempting to print, 
assuming that the file type is recognized. Input Data Type field 350 of Figure 3 indicates 
that the file is an .mp3 file. The user can then specify the data type of up to two outputs 
in Data Type Output fields 352, 356 although in other embodiments, more than two 
outputs can be designated. The menus (not shown) associated with each Data Type 
Output field 352, 356 allow the user to specify among various audio and music formats 
including .mp3, .wav, MIDI, score, transcription and the like. The second output field, 
Data Type Output 2 356 includes a "(NONE)" selection by which the user can indicate 
that she does not want a second output. 

[0047] As shown in Figure 3, the user has selected two outputs, a MIDI file and a 
waveform timeline. The Output Options field 314 also allows the user to designate what 
media it would like each output to be output to, using the Print Output to fields 354, 358. 
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Using pull down menus, the user can select between different choices of output locations 
including memory stored on drives, a print tray, a playback device, an archive, or other 
location coupled to audio processing device 100. In an embodiment, a user can indicate 
that she would like the output to be sent to an email address. When this selection is 
made, an email interface is launched that allows the user to specify the sender and 
recipient email addresses and a text message attaching the output will be generated. As 
shown in Figure 3, the user's choices, entered into the dialog box 300 direct a MIDI file 
version of the input file be output to a CD stored in the D:// drive 354 of the audio 
processing device 100 and a wave form rendering of the input file to be printed to a 
paper document and delivered to print tray 2 358 on audio processing device 100. An 
Indexing Type field 360 is also provided, in which the user can specify how it would like 
an output indexed, in addition to a Time Stamp field 362. As shown in Figure 3, the 
user has selected a bar code index, and does not desire a time stamp to be placed on the 
output. 

[0048] Advanced Options field 310 provides the user with options that are specific 
to the formatting and layout of audio data. In this embodiment, the user selects the 
segmentation type that the user would like to have applied to the audio data. In this 
embodiment of the invention, the user can click on the arrow in the Segmentation Type 
field 316, and a drop-down menu will appear displaying a list of segmentation types 
from which the user can choose. Examples of segmentation options include, but are not 
limited to, segmentation by speaker, melody match, measure, bar, musical line, stanza, 
song, or discrete sound. In the example, the user has not selected any segmentation type 
in the Segmentation Type field 316, so the segmentation type is shown as '*NONE 
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Each segmentation type can have a confidence level associated with each of the events 
detected in that segmentation. For example, if the user has instructed an audio processing 
device 100 to segment the audio file by stanza, each identified stanza will have an 
associated confidence level defining the confidence with which a stanza was correctly 
detected. Within Advanced Options field 310, the user can define or adjust a threshold 
on the confidence values associated with a particular segmentation. 
[0049] In one embodiment, the user can also make layout selections with regard to 
the data representation generated. The user sets, within the "Fit on" field 320, the 
number of pages on which an audio waveform timeline will be displayed. The user also 
selects, within the timeline number selection field 322, the number of timelines to be 
displayed on each page. Additionally, the user selects, within the Orientation field 324, 
the orientation (e.g., vertical or horizontal) of display of the timelines on the multimedia 
representation. For example, as shown in Figure 3, the user can choose to have one 
timeline displayed on one page, horizontally, and this will display the entire audio 
waveform timeline 334 horizontally on a page. As another example, the user can choose 
to have the audio waveform timeline broken up into four portions that are displayed 
vertically over two pages (i.e., two timelines per page). 

[0050] The Preview field 312 shows a preview of the wave form timeline to be 
output to print tray 2 according to the selections chosen by the user. In other 
embodiments, there are two preview fields to represent each of two different outputs. For 
electronic outputs, such as an .mp3 file, a generic representation of the memory medium 
on which the file is to be output, for instance a clip art depiction of a CD, may be shown. 
As shown, the preview includes the number of timelines per page selected by the user 
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(3), and also identifies the name of the file being printed 310 ("Vesoul.mp3"). In 
addition, responsive to the user's choice of a bar code index, the output includes a 
dynamically linked bar code 364 reference to the musical file with which a user can later 
access the file. 

[0051] In the embodiment of Figure 3, there are also shown various buttons, 
including an Update button 326, a Page Setup button 328, an OK button 330, and a 
Cancel button 332. The image of the document shown in Preview field 312 will be 
updated to display any new changes the user has made within print dialog 300. When 
the user selects the OK button 330, the current user-defined preferences are sent to an 
output system to be output. If the user selects the Cancel button 332 at any point in the 
process, the creation of the print job ends and print dialog 300 disappears. 
[0052] Embodiments of the invention involve use of combinations of the modules 
within memory 104 described with reference to Figure 2 to process audio/music data. 
Figure 4 is a flow diagram of steps carried out by a preferred embodiment of audio 
processing device 100 using multiple elements 200-212 to generate the paper output 
depicted in Figure 5. In an embodiment, the steps of Figure 4 are carried out by audio 
processing device 100 of Figure 1 installed with the memory of Figure 2. However, 
other versions of audio processing device 100 with memory as described herein could 
also carry out these steps. The process shown in Figure 4 begins when the audio 
processing device 100 receives 410 an audio file. A user sends the file to audio 
processing device 100 from a networked PC over an Ethernet connection, and it is stored 
to audio/music storage 202. Along with the file, the user sends instructions to generate 
an indexed score based on the audio file over a signal line to audio processing device 
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100 and the instructions are routed to system module 200 over system bus 1 10. System 
module 200 receives the instructions and initiates a series of steps to carry out the 
request. 

[0053] First, system module 200 determines 420 whether the file is a MIDI file. If 
the file is determined not to be a MIDI file, then system module 200, with the help of 
detection module (not shown) determines 422 the format of the file, in this case, an 
audio file in ,mp3 format. The system module 200 sends a command over system bus 
1 10 to conversion module 204 to convert 424 the file from .mp3 to MIDI. Conversion 
module 204 accesses the file over system bus 1 10 in audio music storage 202, and 
creates a MIDI file that approximates the audio file. It sends the MIDI file to system 
module 200, which then stores it to audio music storage 202. If the audio file is a MIDI 
file or has been converted into one, system module activates a user interface module (not 
shown) instructing it to prompt the user for her scoring preferences 432. The user 
interface then sends data signals over system bus 110 representing a dialog box similar 
to the one depicted in Figure 3 to the system module 200 to be output on the user's PC. 
Responsive to the dialog box 432, the user specifies the outputs she would like - a score 
and a MIDI file indexed by measure - and how she would like the output to be presented 
(on paper and burned to a CD) with reference to parameters such as the number of lines 
of music, the style of the notes, the frequency of bar codes, and the format of the bar 
codes. The system module receives the scoring preferences 430, and then stores them in 
audio music storage 202. 

[0054] System module 200 then initiates the scoring process on the 
scoring/transcribing module 208. First, scoring/transcribing module 208 sets up a file to 
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store the score, and assigns 440 a score identifier to the file, for instance a number. 
Scoring/transcribing module 208 then carries out conversion of the MIDI file to generate 
450 a score. Scoring/transcribing module 208 saves the data to the score file and formats 
the score responsive to preferences entered by the user. Scoring/transcribing module 208 
communicates to system module 200 that the score has been completed. System module 
200 then sends the score file information to output system 108 with output instructions 
provided by the user to print the score to a paper document and the document is printed 
460 accordingly. In parallel, system module 200 initiates the generation of the second 
output. It sends instructions to indexing/mapping module 210 to create 470 an index to 
the MIDI file by measure responsive to the score. Indexing/mapping module 210 
accesses the MIDI file and score of the file, both stored in audio music storage 202, over 
system bus 110. 

[0055] Indexing/mapping module 210 determines the beginning of each musical 
measure, based on the score, and creates 470 a measure index to the MIDI file that 
references the beginning and end of each measure. Responsive to instructions from 
system module 200, indexing/mapping module 210 assigns an identifier, for instance, a 
bar code pointer, to each of three measure segments. Indexing/mapping module 210 
then accesses the original score, and maps 480 the bar codes to the score in the 
appropriate locations in the format requested by the user. Indexing/mapping module 210 
decides the appropriate location for the barcodes, using a placement algorithm for 
instance as described in J. S. Doerschler and H. Freeman, "A rule-based system for 
dense-map name placement," Communications of the ACM, v. 35 no. 1, 68-79, 1992. 
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[0056] An exemplary resulting product, a postscript file, is depicted in Figure 5. As 
shown, the melody is divided into four two to three measure segments 510. The score 
indicates that the song is in G major, and dynamic pointers to the end and beginning of 
each segment are referenced by bar codes 520. The bar codes 520 point to specific 
sections in the MIDI file that contains the melody. A two-dimensional bar code 530 has 
also been created by indexing/mapping module 210 and placed in the file that identifies 
the entire MIDI file as a whole, and is output at the bottom of the score for ease of 
reference. In an embodiment, when a user later wants to hear portions of the melody, 
she prints out a copy of the postscript file. She then uses a decoding device (a two- 
dimensional bar code scanner) to access the MIDI data and listen to the selected portions 
of the file. 

[0057] Returning to Figure 4, after the indexed score has been created, 
indexing/mapping module 210 sends a message to system module 200 providing the 
filename of the indexed score. System module 200 sends the indexed score to output 
system 108, and instructs it to save 490 the indexed score dynamically linked to the 
MIDI file to a blank CD stored in a drive of audio processing device 100. At some later 
point, various files used to generate the outputs - including the .mp3 file and portions of 
the score - are marked to be deleted from memory 104. In another embodiment, the first 
measure of the MIDI file 510a, referenced by bar code 520a, is extracted and saved to 
audio/music storage 202. Output system 108 then outputs the short segment to a 
memory card to be inserted into a cell phone and used as a ring tone. In another 
embodiment, audio processing device 100 directly receives two files - the score and the 
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MIDI file - and carries out an abbreviated version of the steps in Figure 4 including steps 
410, 470, 480, and 490. 

[0058] Figure 6 is a flow diagram showing how a portion of a score file stored by 
audio processing device 100 printer could be retrieved and read by an access device. For 
example, a CD contains an archive of musical clips and a barcode index to these clips 
stored in an image file. An access device (not shown) could comprise a standard PC 
with a CD drive coupled to a bar code reader by a signal line. To access the clips, the 
access device would first access the image of the barcode index from the CD in the CD 
drive of the PC 602. A user could print the image for ease of handling to a conventional 
printer coupled to a PC by a signal line. Next, the user locates 604 the relevant bar code. 
Using the bar code reader, the user uses the bar code reader to read the bar code, yielding 
a specific score number and the line number associated with the portion the user wants to 
access. The score with the correct score number (e.g., remapjScoreNo.xml) is loaded 
606, and the line number (e.g., remap_LineNo.xml) associated with the desired clip is 
used to locate the specific line and clip stored on the CD. Once these are located, the 
computer plays 610 the recording, starting with the begin time of the line closest to bar 
code that was scanned. 

[0059] The foregoing description of the embodiments of the invention has been 
presented for the purpose of illustration; it is not intended to be exhaustive or to limit the 
invention to the precise forms disclosed. Persons skilled in the relevant art can 
appreciate that many modifications and variations are possible in light of the above 
teachings. It is therefore intended that the scope of the invention be limited not by this 
detailed description, but rather by the claims appended hereto. 
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